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ABSTRACT 

The purpose of this preliminary report was to provide 
information on the New Approach Method, on program effectiveness 
which can be used for decision-making. The first fifty children, half 
boys, half girls, were tested for attitudes toward Reading- Re la ted 
Activities. Immaturity of children caused a delay in the completion 
of this program, since children take various amounts of time to 
finish, and it was decided that one year was not enough to make 
conclusions. Thus far, general opinion is that gains have been made 
and skills mastered through the NAM program. Parental reaction to NAM 
is discussed and nature of the evaluation explained. (RG) 
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I. INTRODUCTION 
Background 

The first large-scale tryout of the New Approach Method (NAM) , which 
has been described in other papers, has proceeded in Trenton, New Jersey 
for about one full year at the date of this writing. 

Originally, the NAM program was to. have been funded for one year. 

At the end of that period oi: time, it was anticipated that two fairly 
sizable groups of children would have completed the 84 tape recorded 
lessons which comprise the heart of the NAM program. 

Delays 

Unforeseen circumstances, however, have resulted in the completion 
of the program by fewer children than originally expected. The two 
separate six month phases originally planned have merged into one phase 
which has become longer than the combined duration of the two originally 
anticipated six month phases. 

Reasons for the delay in graduating a sizable number of children 
from the NAM program are many. Probl ems of turnover and attrition have 
reduced the number of possible NAM graduates. Unanticipated transfer of 
NAM children to Head Start, kindergarten^ or other programs in the fall 
accounted for much of this unanticipated reduction in sample size. 

Another reason for the relatively small number of NAM graduates to 
date may be tie NAM philosophy itself. Since NAM children are encouraged 
to proceed at a pace at which they feel comfortable, the program necessitates 
different children taking different amounts of time to finish the program. 
Moreover, since the majority of the children who have enrolled at the NAM 
mini**centers are, in general, younger than the smaller number of NAM 
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graduates in earlier programs, the program may have experienced additional 
delays because of the relative immaturity of the children in the present 
study. 

For whatever the reasons, then, NAM personnel have, to date, been 
unable to graduate the number of children originally expected. 

Role of Evaluation 

One of the purposes of evaluation is to provide information on program 
effectiveness which can be used for decision-making. Some of the questions 
which may be based in part or in whole on evaluation findings are the 
following : 

What elements of the program seem to be least effective and, therefore, 
most in need of change? 

Should the program be adopted as a routine procedure in the setting 
in which the evaluation occurs? Should it be expanded to other 
settings? 

Should the present level of program funding be continued? 

These are only a few of the questions for which policy makers may wish 
to base decisions at least partly on evaluation findings. While no 
evaluation report can be so definite as to state that a program should or 
should not be refunded, expanded, or changed, it can present information 
which should allow more rational decisions to be made. 

Funding Crisis 

At the end of fiscal year 1971-72, the NAM program had exhausted its 
funds. A three month extension grant has recently been awarded in order 
that NAM staff might complete instruction for the majority of the children 



enrolled thus far 



Originally, evaluation plans called for only a final evaluation report 
by this office. This final report would contain all of the information 
gathered on program effectiveness when ali (or at least almost all) of the 
NAM enrollees had completed the program. 

However, because of the delays mentioned above and because another 
funding crisis will soon arise, it has been deemed necessary to provide 
preliminary data on the children who have, to date, finished the program. 
Needless to say, this report is being submitted to fulfill the need for 
at least some evaluation findings, even though data is still being gathered. 

Nature of This Evaluation 

Evaluation in the real world often poses a number of problems. Some of 
the most frequently occurring ones deal with the evaluator's inability to 
mainpulate variables, to control for certain factors or to randomly assign 
subjects to appropriate conditions. Many of these problems have occurred 
in the present evaluation. 

Our original proposal called for. the location of a sizable number of 
young children whose parents would be willing to enroll them in the NAM 
program. If a large enough group could be recruited by NAM personnel, as. 
we had originally expected, some could be randomly "withheld" from first 
phase activities and assigned to the second six month phase. Testing these 
second phase children at the appropriate times would have allowed certain 
fairly powerful comparisons to be made. 

The plan mentioned above proved to be infeasible for a number of 
reasons, some of them very practical ones. Since the salaries* of NAM 
learning partners depended on the number of children whom they were able 
to recruit and begin instructing, it seemed unfair to require them to 
withhold children from the program, even in the interest of evaluation. 

Our present hindsight also tells us that had the original plan been 
Implemented, many of the children assigned to the second phase would 
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probably have been unavailable at the start of that phase because of the 
length of time needed to finish Phase I. In any event, a different 
evaluation strategy had to be devised. 

The strategy we now employ involves the use of several evaluative 
comparisons, none of which can alone eliminate all sources of internal 



and external validity, but which together can eliminate most of these 
threats. 

The most powerful technique we are currently using is the random 
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assignment of a group of four-year-olds at Our Lady of the Divine Shephard 
Community Center (O.L.D.S.C.C.) to either a control condition or an 
experimental one (NAM). While we can tell what effects the NAM program 
has produce.d (in conjunction with the routine provided at O.L.D.S.C.C.) 
our ability to generalize to other populations may be somewhat restricted. 

Fop example, we would also like to know what effects NAM has on three- or 
five-year— old children or on children who do not also receive the type of 
program offered at O.L.D.S.C.C. 

In order to shed further light on NAM effectiveness, additional 
evaluative comparisons will be made using data, on children in other NAM 
./ conditions — mini-centers or at-home. These will involve simple comparisons 
of pretest and posttest scores, and more detailed comparisons involving 
pretest scores of some children with posttest scores of other children who 
have been matched on relevant characteristics. Age will certainly be one 
of these characteristics. In addition we have made some use of normative 
data available for the instruments we have used. 

We have also attempted to locate and administer our battery of posttests 
to children who had started the NAM program and were pretested, but who have 
subsequently dropped out of the program. These children should provide some 
useful comparative information, since they were recruited in the same way as 
the NAM enrollees who have remained in the program. If further . information 

shows these dropouts to be similar to NAM graduates on a number of important 
- - — 
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ciutiiicLetxstxcs, ic may De possible to attribute differences in posttest scores 
between the groups ac least partly to differences between the length of their 

participation in the NAM program. At this time this group of dropouts is 

small and hopefully the number of dropouts will remain small. Nevertheless, 

we are prepared to capitalize on NAM’s losses. 

With the combination of techniques discussed above we hope to provide 

some definite conclusions regarding the effectiveness of NAM program. This 

preliminary report will furnish data for some of those techniques. 

Nature of this Report - 

The present report will differ from the final report in a number of 

ways. First of all, the results reported here are based on a much smaller 

number of cases than the results which will appear in a final report. 

Secondly, many of the most important evaluative comparisons planned are 

% 

at this time premature. Comparisons of the pretest scores of children in 
a certain age range at time of pretest with the posttest s cores of another 
group of children in this same age range at posttest time, for example, 
depend on a sample large enough to ensure the required overlap in ages 
from pretest to posttest. At this point, the sample size precludes such 
comparisons. 

One other extremely important comparison .to be made involves the two 
groups of children at Our Lady of the Divine Shephard Community Center. . 

As we mentioned above, one group of randomly assigned four-year-olds is 
currently receiving NAM lessons as a supplement to the regular activities 
of the center, while another group has been randomly assigned to a control 
condition. At .this writing only four children have completed the. series 
of NAM lessons at this center. Comparisons would, therefore, be of little 
value at this time. 

Thirdly, this report will contain little description of the program 
or of its activities, but will focus on the results, i.e., those data which 
shed the most light on program effectiveness. Likewise, tests of statistical 

significance, information on the reliability of our tests, and detailed 
analyses by sex, age level, etc. will be deferred until a-complete set of 
data has been gathered and analyzed. ° 
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II. THE INSTRUMENTS 

In our evaluation of NAM, we have tried to assemble a battery of 
instruments which we felt was a fair, although by no means a comprehensive, 
measure of all NAM objectives. 

Several problems became immediately apparent when we began to consider 
various tests and measures for use in the evaluation. Some of the 
considerations which affected our decisions included the following: 

There was little time for extensive development and tryout of instruments 
specifically designed to measure the objectives of the NAM program. 

Testing time had to be relatively short, both because of the ages of 
the children and because of budgetary constraints. 

Tests- should be easily administered by local residents when given 
suitable training. 

Above all, the instruments should be appropriate for young inner-city 
children. 

Description of the Measures Used 

Several instruments, each individually administered, have been used to 
assess the effectiveness of the NAM program. .One test, which has been 
designated as the "NAM Test," contains items from the Sesame Street Tests 
and also items that were specifically constructed as measures of NAM 
objectives. This test was administered to children on a pretest-posttest 
basis. 

This NAM test contains six sections which are composed of items 
dealing with: 

1. Colors 4. Numbers 

2. Concepts 5. Letters 

3. Shapes 6. Word Reading 
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Subtests 

The Color subtest requires a child to identify basic colors when 
presented with the appropriate stimuli. The Concepts section requires 
a child to demonstrate his understanding of certain relational concepts 
by pointing to the one picture in a set which correctly depicts that 
concept. ("There’s a bear here, here, here, and here. Which bear is the 
smallest bear?") 

The section on Shapes contains several items which require the child 
to name particular shapes when they are presented. ("What is this called?” 
or "What's the name of this?") Other items in this section require the 
child to recognize these shapes. ("Look at this, this, this, and this. 

Which one is a circle?") 

The section on Numbers contains both recognition and naming type items, 
as well as items measuring knowledge of number-numeral correspondence and 
counting. 

The Letters subtest also contains both recognition and naming type 
items, as does the section on Word Reading. 

One other measure which has also been administered on a pretest-posttest 
basis is one consisting of twenty items, ten each from the "Letters and 
Sounds" and the "Aural Comprehension" sections of the Stanford Early School 
Achievement Test (SESAT). Items contained in the "Letters and Sounds" 
section instruct the child as follows: "Look at the box that starts with 

a picture of a candle. The other pictures are car, boat, lid. Point to 
the one that starts with the same sound as candle — car, boat, or lid." 

Items contained in the "Aural Comprehension" subtest require the child 
to demonstrate his understanding of an orally presented story by pointing 
to an appropriate picture after the story is read to him. 
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Sampling items from sections of the SESAT was considered to be both 
desirable and feasible for several reasons. - This sampling allowed us to 
decrease the amount of testing time which would have been needed to 
administer the whole test and to eliminate some of the relatively more 
difficult items. The elimination of these items was considered desirable, 
since the SESAT is normed on children who, on the average, are slightly 
older than the children in our sample. An advantage of the test is that 
individual item statistics are provided in the test's technical manual. 

Thus, comparisons using normative data can be made even though total 
scores are not obtained. 

An attitude measure has also been devised and administered on a pretest- 
posttest basis. This measure, called "Attitudes Toward Reading-Related 
Activities," is being used to detect changes in children's attitudes towards 
reading and reading-related activities. The test first attempts to determine 
if the child understands the difference between ."happy" and "sad" by 
requiring the child point to a picture which indicates how ha feels when 
certain things happen to him. ("Here's Jimmy dropping his ice cream cone. 
Are you happy or sad when you drop your ice cream cone?") 

The next section requires the child to point to the picture that 
indicates how he feels about various reading-related activities. ("Here's 

Jimmy looking at a story book. Are you happy or sad when you look at a 

• • 

story book?") 

The final section of the attitude test requires children to express 
a preference for one of two paired activities, one of which is a reading- 
related activity. ("Here’s a boy listening to a story. Here's a boy 
singing a song. Which would you like to do, or which do you like best?") 
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An additional measure has been given only upon completion of the 
program. That test contains items measuring knowledge of beginning sounds 
("Look at the bear in the next arrow — bear. Point to the letter that bear 
begins with."), as. well as items testing recognition of several words and 
word families frequently presented in the NAM lessons. Finally the child 
is asked to write his name for the examiner. 

The descriptions provided above of the instruments used in our 
evaluation are, admittedly, brief but hopefully they give the reader some 
flavor for the kinds of measures we have used. 
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III. PRELIMINARY FINDINGS 

The results presented in this report are of tyo types - — information 
gathered by administering the tests mentioned above and information from 
responses to questionnaires adminif.tered to parents. 

Problems in Interpreting Data . 

The major comparisons to be made using test data available at this 
point are those looking at differences between pretest and posttest 
scores. By making this type of comparison we can tell if, on the average, 
children are more skilled in certain areas now than they were at the time 
they j/ere pretested. We cannot, however, be certain of how much of this 
change can be attributed to the educational treatment that occurred 
between testing periods, in this case the NAM program. 

Other, comparisons are needed to eliminate other plausible reasons for 
. ....the. changes. Two very plausible reasons for the changes in test scores of 
NAM graduates are those of history and maturation. It is possible that 
some other events or experiences affected NAM children during the time 
presetting and posttesting and that these events or experiences were responsible 
for the changes observed. For example, a child may have learned to 
recognize letters by watching the TV program Sesame Street or by attending 
Head Start classes, etc. It iss also probable that improved performance 
on the measures used is at least partially due to the fact that the child 
is now older and better able to respond to the measures. 

In order to eliminate some of the most plausible rival hypotheses for 
changes in children’s scores, it is necessary to make comparisons in 
addition to the simple pretest-posttest type. It is also ‘desirable to 
determine, if possible, what events have occurred in the period between 
pretesting and posttesting. Were children, in fact, watching Sesame Street . 
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attending Head Start, or participating in some other activity which might 
account for 'imporved test score performance? To provide some answers to 
these questions we have included in our parent questionnaire certain 
questions seeking information on the experience and habits of the NAM 
children . 

Description of Children in the Sample 

The children on whom the data in this preliminary report are based are 
the first 50 children to have finished -he NAM program. Exactly half are 
boys, half are girls. 

Only two of the children in this sample have worked with their own paents 
at home, while the remaining 48 have received the NAM lessons at one of four 
NAM mini-centers. Nearly half (23) of the mini-center graduates have come 
from one of the four centers. 

The median age of the children when they were pretested was 56 months. 
Almost 95% of the children were between 45 and 70 months old at that time. 

The median age at time of posttest was about 64 months with 95% of the 
children being between 54 and 78 months old. The median time between pretest 
and posttest, which reflects the length of time taken to finish the program, 
was 8 months. One child was able to finish the series of lessons in less 
than 4 months while three others took almost 11 months. 

When more data becomes available we will make comparisons to determine 
whether younger children take longer to finish the lessons thin do older ones. 
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Discussion of Test Results 



Test results for the 48 children having both pretest and posttest 
scores are presented by subtest in Table 1. The overall picture shows that 
this group of children gained in each of the areas tested. Some of the 
gains are dramatic; others are less pronounced. Since no tests of 
statistical significance will be provided in this report, small gains 
should be viewed cautiously. The reader is cautioned that although we 
may speak of small gains throughout this report, some of them may actually 
be non-significant. 

Inspection of Tables 1 and 2 reveals that children scored relatively 
high on several of the pretests. Very large gains from pretest to posttest 
then, were not possible on these subtests. On some other tests posttest 
scores were quite high, reaching the maximum possible score in several 
instances. This ceiling effect also may have depressed gain scores. 

There were other subtests, however, on which the opposite was true. 

The fact that initial scores were quite low in some instances would allow 
instructional effects to be readily detected. Discussion of results by 



subtest follows. 



Table 1 



13 



Pretest, Posttest, and Gain Scores for All Children 
Having Both Pretest and Posttest Scores (N = 48) 



Subtest 


. Maximum 
Possible 
Score 


Pretest 
Mean SD 


Posttest 
Mean SD 


Gain 

Mean SD 


Colors 


4 


2.6 


1.5 


4.0 


0.0 


1.4 


1.5 


Concepts 


10 


8.0 


1.3 


9.1 


0.8 


1.1 


1.5 


Shapes 


7 


4.1 


2.1 


6.4 


1.1 


2,3 


2.0 


Numbers 


10 


5.2 


2.7 


9.4 


0.8 


4.2 


2.5 


Letters 


16 


5.7 


4.2 


15.2 


1.6 


9.4 


4.1 


Word Reading 


18 


2.7 


2.2 


8.4 


4.9 


5.6 


4,6 


Counting 


30 


13.9 


9.7 


25.7 


6.6 


11.9 


9.6 


Letters and Sounds* 


10 


4.5 


2.0 


6.5 


2.8 


2.0 


3.1 


* 

Aural Comprehension 


10 


6.0 


2.0 


7.4 


1.7 


1.4 


1.8 


Beginning Sounds** 


7 


*** 


5.3 


3.4 


*** 


** 

Sight Words 


3 


*** 


1.7 


1.2 




*** 


** 

Word Families 


5 


*** 


1.7 


3.4 




*** 



A 

N = 46 for these subtests 

"/cfc 

N ■ 50 for these subtests 
*** 



Posttest only 



Table 2 
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Percentage of NAM Children Answering Each Item 
Correctly at Pretest and Posttest 1 



COLORS 

Naming : 

1. Red 

2. Blue 

3. Green 

4. Yellow 

CONCEPTS 

5. Which ball is same? 

6. Which pencil is longest? 

7. Which is straight line? 

8. Which is biggest bear? 

9. Which is smallest bear? 

10. Which bird is above cage? 

11. Which dog is in box? 

12. Which trees are all in row? 

13. Which balloon is at bottom? 

14. Which balloon is at top? 

SHAPES 

What is this called: 

15. Square 

16. Circle 

17. Rectangle 

18. Triangle 
Which one is a: 

19. Circle 

20. Square 

21. Triangle 



Pretest % Post test % 

Correct Correct 



N=48 


N=50 


68 


100 


50 


; 100 


68 


100 


64 


100 



94 


100 


92 • 


100 


92 


98 


90 


100 


86 


94 


32 


46 


92 


100 


48 


74 


74 


100 


72 


100 



46 


90 


74 


98 


16 


76 


46 


90 


82 


98 


72 


94 


62 


94 



small 
a larger 



The reader is urged to remember that these estimates are based on a relatively 
sample and are, therefore, subject to more fluctuation than estimates based on 
sample would be. ig 
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Pretest % Posttest % 

Correct Correct 

N-48 N=50 



NUMBERS 



22. 


Which is: 
3 




70 


100 


23. 


8 




62 


100 


24. 


What is this: 
6 




30 


98 


25. 


9 




28 


90 


26. 


2 




46 


. 100 


• 

r^. 

CN 


10 




20 


88 


28. 


Number/Numeral Correspondence: 
2 frogs 




70 


100 


29. 


5 turtles 




58 


100 


30. 


Numerical relations: 
First 




78 


100 


31. 


Last 




36 


64 


32. 


Counting: 

To 10 without mistakes 




5.2 


100 


33. 


To 20 without mistakes 




24 


78 


34. ’ 


To 30 without mistakes 




18 


54 


LETTERS 

Which letter is: 
35. A 


* 


54 


100 


36. 


P 




40 


98 


37. 


B 




56 


100 


38. 


E 




62 


.. 100 


39. 


f 




48 


100 


40. 


b 




36 


78 


41. 


n 




34 


96 


42. 


h 




42 


96 



16 



It 



LETTERS (continued) 

What is this letter: 



43. 


S 




44. 


C 




45. 


H 




46. 


W 




47. 


m 




48. 


e 




49. 


t 




50. 


g 




READING WORDS 






Which one 


says; 


51. 


BIRD 




52. 


SUN 




53. 


nop 




54. 


BOAT 




55. 


AT 




56. 


sister 




57. 


hand 






What does 


this say: 


58. 


TO 




59. 


HAT 




60. 


STREET 




61. 


met 




62. 


mouse 




63. 


big 

What does 


this sentence say: 


64. 


THE 




65. 


LITTLE 




66. 


BOY 




67. 


IS 




68. 


HAPPY 





Pretest % 
Correct 
11=48 



32 

26 

28 

28 

20 

20 

14 

10 



38 

26 

16 

14 

22 

32 

42 

10 

10 

6 

6 

4 

6 

6 

6 

10 

2 

4 



Posttest % 
Correct 
N=50 



100 

94 

92 

86 

96 

94 

100 

88 



78 

82 

64 

64 

80 

90 

64 

42 

38 

12 

32 

38 

26 

26 

16 

34 

40 

14 



O 

ERIC 
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LETTERS AND SOUNDS 

(Stanford Early School Achievement Test) 
Sound of : 



Pretest % 
Correct 
N=48 



69. 


g 




40 


70. 


m 


* 


38 


71. 


c 




42 


72. 


t 




62 


73. 


r 




30 


74. 


P 




40 


75. 


h 




26 


76. 


s 




40 


77. 


d 




48 


OO 

• 


sh 




46 


AURAL COMPREHENSION 

(Stanford Early School Achievement Test) 






79. 


Story 1 




56 


80. 


Story 2 




62 


81. 


Story 3 




70 


82. 


Story 4 




70 


83. 


Story 5 


• 


48 


84. 


Story 6 




62 


85. 


Story 7 




30 


86. 


Story 8 


• 


52 


OO 

• 


Story 9 




60 


OO 

OO 

• 


Story 10 




46 





Posttest 7. 
Correct 
N=50 



72 

76 

72 

68 

60 

70 

64 

64 

56 

58 



80 

90 

86 

78 

64 

62 

68 

72 

84 

46 



BEGINNING SOUNDS 
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Pretest % Posttest 7 , 

Correct Correct 

N=48 N=50 



Which letter (s) does this word being with: 



89. 


Bear 


*** 


84 


90. 


Sandwich 


*** 


86 


91. 


Apple 


*** 


78 


92. 


Pan 


*** 


80 


93. 


Foot 


*** 


74 


94. 


Wig 


*** 


78 


95. 


Thumb 


*** 


. 46 

: 


SIGHT WORDS 






96. 


BUS 


*** 


72 


97. 


CAR 


*** 


58 


98. 


BIKE 


*** 


40 


WORD FAMILIES 






99. 


(P)AM 


*** 


50 


100. 


(P)AT 


*** 


40 


101. 


(P)ET 


*** 


28 


102. 


(P)AY 


*** 


18 


103. 


(P)AN 


*** 


38 



* :** 



Post test only 
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Colors : The colors subtest was the first test that the child received. 

It was thought that children would generally be more familiar with basic 
colors than with some of the other areas that were to be tested later 
in the testing sequence. This section, then, was used not only to determine 
if the exercises used in the- NAM program had any effect on the child's 
knowledge of colors, but also to help the child ^ e 9 oine a c 9 ustoae d to .. . 

the testing situation, which may very well have been his first such 
experience. 

Pretest scores indicated that children tended to be 
relatively familiar with the basic colors when they started the program. But 
the posttest scores of the first 50 NAM graduates should be quite pleasing to 
NAM personnel, for every child was able to name every color on the 
posttest. 

Concepts : Knowledge of ten relational concepts stressed in the first 

few NAM lessons and reviewed in later lessons was tested in the concepts 

subtest. Again, the data we had indicated that these items should also 

be relatively easy for children in this age group. This proved to be the 

case since, on the average, children were able to answer 8 of tue 10 pretest 

items correctly, indicating that they were, in fact, capable of responding 

/ 

appropriately. Although pretest scores were high, a slight gain occurred 
from pretest to posttest, pushing scores near the maximum possible score. 

Table 2 reveals differences in the relative difficulties of the 
items. The concept of "above" was the most difficult concept when 
children were first tested. It also tended to be quite difficult on the 
posttest, as was the concept of "all in a row." On the whole, by posttest 
time, children seemed to have mastered all the other concepts tested. 



Shapes : The NAM children also improved with regard to their ability 
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to name and recognize simple shapes (circle, triangle, square and 
rectangle). Posttest scores again were almost as high as possible. 

Table 2 shows that children tended to be better able to recognize or 
name a circle at both pre- and posttest than any of the other shapes. 

However, at posttest time children had become familiar with all the shapes we 
presented, the rectangle still being the most difficult. 

Numbers : The numbers subtest contained items testing both recognition 

and naming one and two digit numbers. In addition, there were two items 
dealing with number /numeral correspondence and two others testing 
knowledge of the concepts "first" and "last," although these latter two 
items might also have been included in the concepts subtest. 

Again gains were quite high for this section. The pretest 
mean of 5.2 increased to 9.4 (of 10) on the posttest. Table 2 indicates 
that only the concept of "last" remained very difficult for these children. 

Letters : Probably the most dramatic gains from pre- to posttest 

occurred on the letters subtest, which contains eight items requiring 
recognition and eight items requiring naming of letters. A mean gain of 
over 9 points resulted in near perfect posttest. scores for most of the 
children in our sample. 

Although it is desirable to look at both recognition and 
naming scores for this subtest, that data, unfortunately, was not 
available at this writing but will be provided in a final report. 

Table 2 reveals that, in general, these 50 children found 
the tasks of naming or recognizing both upper and lower case letters to 
be quite easy on the post test. Percentages presented in Table 2 show that 
naming letters was more difficult on pretest than was recognizing letters, 
even after adjustment for guessing. On the posttest, however, the children 
in our sample were able to perform both tasks about equally well. 

The only item on the posttest which was at all difficult 
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was that one requiring recognition of lower case "b." The fact that this 
item contains a lower case "d" as one of the distractors may account for 
the relative difficulty of the item. 

Word Reading : The word reading subtest, like several previous 

subtests, contains both recognition and naming type items. Recognition 
required the child to pick out the appropriate word from a set of four 
when he was told that word (and in some cases also given a picture of 
that word). Naming required the child to actually read certain words. 

When children began the NAM program very few were 
able to read or recognize many of the words presented. In fact, the 
pretest 'average of 2.7 was not much higher than the average score (1.8) 
that the group would have received by chance alone. In contrast, at 
posttest NAM graduates were able to answer nearly half (8.4) of the 18 
items correctly. 

As with the letters subtest, we were unable to get 
score breakdowns by recognition and naming in time for this report but 
these scores will most certainly be provided in the final report. Table 
2, however, suggest that gains were again more pronounced for recognition 
items than for those items requiring the child to make a verbal response. 

Counting : The children in our sample seemed to be able to count 

fairly well at the time of pretesting. On the average, the children 
could count to about 14 at that time, while on the posttest they were 
able to count nearly 12 digits further. 

Table 2 shows that all of the children posttested to 
date were able to correctly count to 10 on the posttest and over half 
of them were able to count to 30 without making any mistakes. 
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Letters and Sounds (Stanford Early School Achievement Test) ; The 
children in our sample showed moderate gains on the 10 items selected 
from the Letters and Sounds subtest of the Stanford Early School 
Achievement Test. The selected items require the child to point to the 
picture of the object that starts with the same sound as another object 
(e.g., gate starts with the same sound as rose, gift or witch) after the 
child is told the name of each object. 

Table 2 reveals that all items were less difficult for these 
children when they had finished the NAM program than when they had just 
started. As was previously stated, one advantage of the Stanford Early 
Sctiool Achievement Test, at least for our purposes, is that individual item 
statistics are provided for a large national sample. These norms are given 
for groups at the beginning of kindergarten and beginning of first grade. 

Table 3 presents the median percentages for the two subtests 
composed of items selected from the Stanford Early School Achievement 
Test (SESAT) for both the NAM sample and for the group on which national 
norms were established. 

Table 3 

Medians of Percentages Answering Each Item 
• . Correctly for the Letters and Sounds Subtest 

of the SESAT 



Pre Post 



NAM Sample 


.40+ 


. 66 * 


SESAT Norming Sample 


* 

.39 


** 

.64 



+ N = 46 ++ N “ 50 



Beginning kindergarten 
Beginning first grade 
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Table 3 indicates that the set of items composing the 
letters and Sounds subtest was about equal in difficulty for NAM 
children at pretest and a national sample of children beginning 
kindergarten. The posttest median for NAM children is again virtually 
the same as that for a national sample of children beginning first grade. 

One important point which must be made here is that all of 
our testing was done on an individual basis. The national norms for 
the SESAT were based on the results of group administrations. The 
differences in item statistics that the two procedures may produce is 
uncertain. One would suspect that somewhat higher scores might result 
from individually administered tests, although the exact effect is 
uncertain. 

Nevertheless, if we can attach some credibility to our 
findings, we would have to say that the NAM sample, although somewhat 
younger than the national sample, started out with about the same degree 
of skill (as measured by these items) as the national sample. Moreover, 
the NAM group seemed to maintain pace with the older group and they did 
so in a somewhat shorter period of time, since less than a year has 
elapsed from the time the first NAM enrollee was pretested to the time 
the last child in the present sample was posttested. 

Aural Comprehension (SESAT) : Table 1 reveals that a modest gain 
was achieved from pre- to posttest on this section of our test battery. 
This measure of comprehension was included in the battery to determine 
if improved aural comprehension might be a side-effect of the NAM 
instructional program. Since the program requires the child to listen 
to and react to tape recorded instructions, we postulated that 
improvement in that area might occur. 





Table 4 suggests that the NAM children were somewhat 
less able to answer these questions at pretest than were the children 
comprising the SESAT norming sample. The gap at posttest remained about 
the same, indicating that NAM graduates seem to have again kept pace with 
the children in the national sample. 

Table 4 

Medians of Percentages Answering Each Item 
Correctly for the Aural Comprehension 
Subtest of the SESAT 



Pre Post 



NAM Sample 


.58 + 


.75+* 


SESAT Norming Sample 


* 

.66 


** 

.84 



+ N = 46 ^N « 50 



Beginning kindergarten 
Beginning first grade 



Beginning Sounds : In contrast to the letters and sounds section of 

the SESAT, this test required the child to choose the letter that a word 
begins with ("Look at the picture of the bear. Point to the letter that 
bear begins with."). The letters and sounds section required the child 
to determine which beginning sounds were the same, without having to 
attach the appropriate letter to that sound. 

This test was administered on a posttest basis 
only. The children seemed to do quite well on these items, averaging 5.3 
of a possible 7. Since children were not very familiar with the alphabet 
when they began the program, as revealed by the relatively low pretest scores 
on the letters subtest, we can probably safely assume that children generally 
would not have been able to match a letter with its sound had the present 
test been given at pretest. Most likely, scores would have hovered around 
a chance score of 1.8. ^5 



a chance score of 1.8 



Table 2 suggests that only the task of attaching 
the "th" sound to the word "thumb" was very difficult for this sample. 

It is interesting to note that the two easiest items for this group were 
those dealing with the sounds of "b" and "s". These sounds were the 
first to be introduced and the ones most frequently repeated in the NAM 
lessons. The "th" sound, on the other hand, was introduced near the end 
of the series of NAM lessons and was, therefore, presented less frequently 
as was the sound of "w." Overall it seems significant that children 
performed so well on this subtest, since the NAM lessons place heavy 
emphasis on a phonics approach. 

Sight Words : On a posttest basis only, children were asked which 

of three words they could identify. These words were ones presented with_ 
varying degrees of frequency in the NAM lessons. 

Almost three quarters of the children could read the 
word "Bus," while about 40 percent knew the word "Bike." These findings 
are, in general, consistent with those of the previously discussed word 
reading segments, i.e., recognition of words is easier than reading them. 

Word Families : The purpose of this section, which was given at 

posttest only, was to determine if children were becoming familiar with 
the word families that were presented in the NAM lessons. 

First our testers tried to make sure that the child 
being tested knew the sound of "p". (Results of the beginning sounds 
subtest indicate that 80 percent could already match the letter "p" with 
the "p" sound in pan.) Several words each beginning with "p" were then 
presented individually. 
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The results shown in Tables 1 and 2 suggest that 
this series of tasks was quite difficult for our sample. Again, however, 
the relative difficulty of the items seems to be related to the 
frequency of presentation in the NAM lessons. Lessons dealing with 
the -am or —at families, for instance, occur much more frequently than 
those presenting members of the -et or -ay families. Our findings 
suggest that children are more familiar with the former families than 
with the latter ones at time of posttesting. We hope that a more formal 
content analysis of the NAM lessons will shed additional light on the 
relationship between frequency of presentation and achievement. 




At/ f 






27 . 



Sesame Street Test Norms 

Embedded in our NAM test battery were several items which were used in 
ETS's evaluation of the first year 0 f the television show Sesame Street . Only 
those items testing skills specifically taught in the NAM lessons were 
included in our battery. 

Since the Sesame Street tests had been shown to be appropriate for 
three-, four—, and five-year-old children from a variety of populations 
(four-year-old disadvantaged children from inner-city areas were the most 
heavily represented group) and since we had available a considerable amount 
of data on these items, we decided to use as many of the Sesame Street items 
as possible. 

The actual number of these items that were judged to measure skills 
taught in the NAM lessons, however, turned out to be relatively small. 
Nonetheless, we felt that the data on these items, might provide useful 
baseline information for our evaluation of the NAM program. • 

There are several important points to remember when one compares the 
Sesame Street data with the data obtained on the present sample of NAM 
children. Although the children in both samples were approximately the 
same age. at pretest, the time between pre- and posttesting was six months 
for the Sesame Street sample. Since the NAM lessons allow children to 
proceed at individual rates, the amount of time between pre— and posttesting 
varied for each child, since we tested each child when he began and again 
when he completed the program. 

For the children in our sample of 50 NAM graduates, the median 
time between pretesting and posttesting was 8 months, although it 
ranged from 3-1/2 to 11 months. Not only were there differences in intervals 
between pre- and posttesting for the two samples, but this interval may have been 
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The median age of the Sesame Street sample was 53 months at pretest. About 

78% of the total sample were termed disadvantaged as determined by traditional 
SES measures. 
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shorter for the more able NAM students than for the less able ones In 
our sample. Needless to say, these conditions restrict the degree of 

comparability between the two sets of data. Nevertheless, data for the 
Sesame Street sample will be presented here to give the reader some idea 
of the difficulty of the items for a large sample of children from five 
geographically dispersed sites. 

It is important to mention here that the presentation of data from the 
Sesame Street evaluation is in no way intended to allow comparisons between 
the effectiveness of Sesame Street and the NAM program. The data is intended 
only to help the reader establish some general perspective for the status 
of NAM children in the absence of more appropriate comparisons. 

Table 5 provides item statistics for those items common to both the 
Sesame Street and the NAM evaluations. The posttest statistics from 
the Sesame Street evaluation presented here reflect what may be considered 
an "average"^ amount .of Sesame Street viewing. Pretest item statistics 
for the Sesame Street sample naturally do not reflect the effects of 
Sesame Street viewing since pretesting was concluded before the start of the 
first year’s telecast. 

It is probably safe to assume, however, that both the pre- and posttest 
NAM results may reflect the effects of Sesame Street viewing. Moreover, it 
will be a difficult if not an impossible task to unravel these effects. To 
complicate matters further, this past year saw the introduction of a new 
television show. The Electri c Co mpany » which, although its primary target 

is children in second grade, may also have had some effect on children in 
our sample . 
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In the first year’s evaluation of Sesame Street children were retroactively 
grouped according to the frequency with which they had watched Sesame Street 

during the preceding year. Quartiles were established in which children 
had viewed the show rarely or never, about 2 or 3 times a week, about 4 or 5 
times a week, and more than 5 times a week. The statistics we have presented 
here are the averages computed using the groups watching 2 or 3 times a week 
and 4 or 5 times a week. Our decision, therefore, which is admittedly <r> 

somewhat arbitrary, may be thought of as representing a group which viewed **' 
on the average 3 or 4 times a week. 



Table 5 



Percentages of NAM Subjects and Sesame Street Subjects Answering 
Each Item Correctly on Pretest and Posttest for Items Common to 
Both Evaluations 
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Pretest ' 


l Correct 


Posttest 


% Correct 






Sesame 




Sesame 


* • 


NAM 


Street 


NAM 


Street 


l • 


N=48 


N=943 


N=50 


N=943 


CONCEPTS 










1. Biggest 


90 


96 


100 


99 


2. Smallest 


86 


81 


94 


93 


3. First 


78 


* 


100 


90 


4. Last 


36 


* 


. 64 


41 


SHAPES 










What is this called: 










5. Square 


46 


29 


90 


61 


6. Circle 


74 


63 


98 


83 


7 . Rectangle 


16 


13 


76 


39 


8. Triangle 


46 


33 


90 


67 


Which one is a: 










9. Circle 


82 


84 


98 


94 


10. Triangle 


62 


33 


94 


67 


NUMBERS 








• 


What is this: 










11. 2 


46 


19 


100 


53 


12. 6 


30 


12 


98 


38 


13. 9 


28 


7 


90 


36 


14. 10 


20 


12 


88 


40 


LETTERS 


. 


' 






■ Which letter is: 










15. A 


54 


38 


100 


68 


16. P 


40 


29 


98 


61 


17. f 


48 


26 


100 


51 


What is this letter: 




• 






18. S 


32 


12 


100 


39 


19. C 


26 


13 


94 


' 38 


20. H 


28 


10 


92 


37 


21. W 


28 


7 


86 


46 


22. m 


20 


6 


96 


28 


23. e 


20 


7 


94 


30 


24. t 


14 


9 


100 


35 


25. g 


10 


2 


88 


11 


READING WORDS 










26. HAT 


10 


1 


38 


2 


27. STREET 


6 


0 


12 


2 


30 









At the present time our data on the Sesame Street viewing habits of 
thfe children in the NAM sample under consideration here is meager. Responses 
from the 25 parent questionnaires that have been returned to date s ee m to 
support our decision to use item statistics for a group viewing Sesame 
Street an "average" amount of time, since parents reported that these 
children either now watch S esame Street or had watched it in the past. 

Table 6 presents parent responses to questions about their children's 
Sesame Street viewing habits. 



Table 6 

Parent Questionnaire Responses to Questions on Sesame 
Street Viewing Habits 

Does your child ever watch the TV show Sesame Street ? 

No .. 

YeS, every day or almost every day 

Yes, about 3 or 4 times a week. ’ 4 

Yes, about 1 or 2 times a week 7 

Yes, less than once a week. _ q 

Don't know.’ • • 1 ; . . n 

Did you child ever watch Sesame Street in the past? 



Yes 22 

No 2 



Don't know 1 

Inspection of Table 5 reveals that, in general, the children in our 
sample were more able to answer the questions used in the Sesame Street 
evaluation at pretest time than were the children in the Sesame street sample 
The exact reason for the relative superiority of NAM children at pretest is 
uncertain. It is possible that it may not be appropriate to consider these 
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children disadvantaged, at least in the educational sense. It is also 
possible that the first 50 children to finish the NAM program are more 
able than the children who are taking longer to complete the program. 
Another possibility,’ as was stated above, is that the effects of previous 
Sesame Street viewing may be at least partly responsible for the relatively 
high pretest scores of our sample. 

Gains in percentages answering each item correctly are, in general, 
also higher for NAM graduates than for the Sesame Street sample. At this 
time we can only speculate as to what agents are responsible for the 
differences in gains observed for eaoh group. Undoubtedly, a combination 
of factors is responsible which we will attempt to unravel as more data 
becomes available. In any event, our posttest results for the children in 
our NAM sample compare favorably with those of the Sesame Street sample. 
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Attitudes Toward Reading-Related Activities 

This test was designed to assess both children's attitudes toward 
reading and related activities and also changes that might occur from pre— 
to posttest. Unfortunately, because of the extremely short time available 
for instrument development when our evaluation began, this measure was not 
available when the first children were pretested. Consequently, we have 
both pre- and pcsttest data on only 15 children at this time. We will, 
therefore, defer discussion of observed changes in attitudes until the final 
report is prepared. 

It may be useful, however, to discuss data for selected posttest items. 
Table 7 suggests that the sample of children taking the attitude measure 
was able, in general, to respond appropriately to the items on the posttest. 
Indeed, all of the children were able to recognize a happy face and almost 
all knew which face was sad or unhappy. 

Section 2, Attitude Towards Reading-Related Activities, of this 
instrument requires the child to point to the picture that shows how he feels 
when he is engaged in certain activities. In general, these children tended 
to be relatively happy regardless of what activity we asked them about. 

Getting a present, whether it was a book or an article of clothing, seemed 
to make children slightly more happy than any of the other activities presented. 
Whether these relationships will hold up using the more stable estimates of a 
larger sample and whether there is any change in attitudes towards particular 
activities from pre- to posttest remains to be seen. 

Section 3 of our attitude measure requires the child to indicate his 
preference for one of two paired activities, one of which is a reading-related 
activity. This section was included in this measure, since data available 
to us regarding the use of items such 



as those found in the previous section 
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Table 7 



Percentage of Children Answering Each Item Correctly 
and Percentage Choosing Each Activity on Posttest (N = 50) 



TEST: ATTITUDES TOWARD READING-RELATED ACTIVITIES 

Happy and Sad : 

1. Which is happy? 

2. Happy or sad? 

3. Opening a present? 

4. Falling off bike 

5. Dropping lollipop 

6. Eating ice cream 

Attitude towards reading-related activity : 

7. Looking at a picture book 

8. Learning ABC's 

&. Watching television 

10. Listening to tape recorder 

11. Drawing and coloring picture 

12. Looking at s£ory book 

13. Singing a song 

14. Getting book as a present 

15. Getting shirt. or dress as present 

16. Listening to story 
Like to do best : 

17. Eat ice cream 
Drink, water 

18. Look at picture book 
Look at story book 

19. Watch television 
Listen to tape recorder 

20. Listen to tape recorder 
Draw and color picture 

21. Get a new book 

Get a new shirt (dress) 

22. Listen to story 
Watch television 

23. Look at story book 
Watch television 

24. Sing a song 

Listen to story 34 



% Correct % Incorrect 



100 


0 


96 


4 


90 


10 


80 


'20 


86 


14 


94 


6 


Ha ppy. 


Sad 


84 


16 


86 


14 


90 


10 


82 


18 


88 


12 


09 


16 


90 


10 


94 


6 


98 


2 


92 


8 



% choosing each option 

78 

22 

64 

36 

• • 

48 

52 

36 

64 

16 

84 

46 

54 

56 

44 

78 

22 
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indicated that children in this age range tended to indicate they were happy 
regardless of the activity, even falling off a cicycle. Section 3 was 
designed to get another "fix" on children's attitudes. 

Table 7 suggests, that children knew what was required of them on this 
section. A test item on which the child expressed a preference for either 
an ice cream cone or a glass of water revealed that children had clear cut 
preferences for ice cream, as we had hypothesized. 

Since baseline data from pretest and from control sources is scarce 
at this time, we will defer discussion of these results until the final 
report is prepared. 

However, there are two items which may have some meaning in the absence 
of comparison data. Those are the items which pair the activity of "listening 
to the tape recorder" with either "watching television" or "coloring a 
picture." Results suggest that children enjoy working with the tape recorder, 
at least as much as watching television, but not as much as drawing and 
coloring. If more stable estimates corroborate, these findings, then the use . 
of tape recorders would indeed seem to be, as NAM originators have contended, 
an effective way of "turning kids on." The fact that children still indicate 
a fairly high preference for the tape recorder after having worked with it 
almost daily for a relatively long period of time seems significant. We 
must note here that it will also be important to look at these statistics 
for pretest results when more data becomes available. 
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When asked about the benefits that their children received from the 
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NAM program (Question 18) , parents indicated that, of those listed on 
the questionnaire, learning to read and learning to enjoy reading and 
related activities were the major benefits. Sixteen of 18 parents 
marking one of the options said that their children definitely began to 
read; 17 of 20 indicated that their children definitely learned to enjoy 
reading and related activities. Table 8 shows the number of parents 
choosing each option. 

The next most frequently perceived benefit, although mentioned only 
about half as often as the benefits discussed above, was a better knowledge 
and understanding of one's own child. Eight responses indicated this to be a 
"definite benefit, while 6 others indicated that 'it was somewhat of a benefit. 

Only four parents thought that the program definitely brought them 
closer to their children, while 7 others thought that the program helped 
somewhat in this regard. When more data becomes available on the children 
working directly with their own parents at home, it will be interesting to 
compare the responses of parents who have worked with their own children 
with the responses of those parent's whose children have attended mini-centers 
It is possible that those parents working more directly with their own 
children would be more likely to notice changes in relationships with their 
children. 

The 25 parents responding to our post questionnaire have given the 
NAM program high success ratings. With regard to the area of teaching 
reading, 18 parents considered the program to be "very successful," 
while 5 others thought it was at least"somewhat successful'.' Likewise, 

18 parents' also considered the program "very helpful" in other areas. Only 
one parent thought the program was "not very helpful" in areas other than 
learning to read. The areas in which parents considered NAM to be helpful 
are presented in Table 9. Again reading was mentioned most often (8 times) 
as an area of benefit. Other areas which are related to the content of the 
NAM lessons » which were also mentioned quite frequently ,were letters 38 
and sounds (5 times) and cnunHnff nr nnmXor arHm'Hac 
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With respect to non-cognitive areas, parents mentioned sharing, 
helping, or cooperating most often (7 times). Six parents thought that 
either their children's behavior had improved or that their children were 
more self-disciplined as a result of their participation in NAM. 

Improvement in communication, understanding, or self-expression was 
mentioned 5 times. Several less frequently mentioned areas are also 
presented in Table .Sa- 
lable 9 

Responses to Question 21 from Parent Post Questionnaire 



Area Frequency of Mention 

Reading 8 

Spelling 3 

Learning Shapes and Sizes 2 

Learning Colors 2 

Letters or Sounds 5 

Coloring and Drawing 2 

Counting or Number Activities 5 

Behavior or Self-discipline 6 

Overcoming Shyness 3 

Sharing, Helping, or Cooperating 7 

Communication, Understanding, or Self- 

Expression 5 

Better Relationships with Others 5 

Independence 4 

Enjoyment of Learning or Interest in Books 2 



In order' to assess overall parent satisfaction with the NAM program 
we also posed the following hypothetical question to parents: "If you 

have other young children, would you also want them to attend the NAM 
program?" This seemed to us to be tantamount to saying, if you had it to 
do over again, would you do it again? Of the 21 parents marking a response 
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to this item, 20 answered ’’yes" while the other said "don* t know." Indeed 
parents seemed to be quite satisfied with the program. 

The last question on our parent questionnaire gave parents a chance 
to make any other comments they might wish to make about their child's 
participation in the NAM program. Since the number of questionnaires returned 
to date is small, we have reproduced all of the parent responses to this 
question, many of them verbatim. These responses need little interpretation 
and are presented in Table 10. ' * 

Finally, we asked children (on the posttest) to endorse our evaluation 
efforts by signing their names for us. All 50 children were able to do 
so. A random sample of these has been reproduced as Table 11. 



Table 10 
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Comments about Participation in NAM Program 
(Question 29) 

Four parents expressed desire for some type of follow through activities. 

One parent expressed concern about children being bored with traditional school 
activities as a result of participation in NAM. Would like to see the program 
open up to every child. 

"The program is beautiful and my child really loved it." 

"NAM has been very helpful to my child and I would recommend it for any child." 
Child enjoyed NAM. • 

"Glad that I learned about the NAM program because it helped my child and me. 

Keep up the good work." 

"NAM is one of the most sincere programs we have been involved with. Parents 
invited to participate and are supported in their efforts. No matter how 
little education they have there is a way one can help." 

"NAM has done wonders for. my son." 

Loving relationship between child and "wonderful" learning partner was responsible 
for child's success in program. 

"I think it is a very good program and has really prepared my child for school 
in every way possible. 7 

The kids and teachers helped child overcome shyness. "Wonderful program for 
preschoolers." 

"She enjoyed the program and still wants to do her lessons." 

"NAM program is wonderful." 

It starts children off with the idea that school is more than finger painting, 
coloring, and playing games." * 

There have been more things learned than I can name." 

e 

"Helped my son in all areas." 

t ... 

"I am very proud [of] what the NAM program has taught my child." 
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Responses to Question 21 which were unclassified. 



Table 11 
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IV. SUMMARY AND CONCLUSIONS 

Unfortunately, the conclusions we can make at this time regarding 
the effectiveness of the NAM program are quite tentative for a number 
of reasons. The' relatively small number in our sample, the relative 
lack of data on some of the more important comparison groups in the 
study, the important differences that may exist between "early program 
finishers" and "late program finishers," and several other factors 
render our findings inconclusive at this stage. . 

The overall picture which the data presented thus far seems to 
paint is that the children in the sample under consideration here have 
made progress, and indeed seem to have mastered several of the skills 
that we have tested. In general, it seems unlikely that gains as 
dramatic as many of those we have observed could have resulted without 
some formal instructional program. Obviously, the NAM program must be 
considered as a likely cause of these gains or, mere appropriately, 
as a likely contributor to these gains. 

With that final observation, we conclude this preliminary report, 
eagerly anticipating more data for our final report, which will include 
the additional information that such reports usually provide. 
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